Introducing real-time lighting effects to illuminate real-world physical objects in see-through augmented reality displays

ABSTRACT

Embodiments provide for the rendering of illumination effects on real-world objects in augmented reality systems. An example method generally includes overlaying a shader on the augmented reality display. The shader generally corresponds to a three-dimensional geometry of an environment in which the augmented reality display is operating, and the shader generally comprises a plurality of vertices forming a plurality of polygons. A computer-generated lighting source is introduced into the augmented reality display. One or more polygons of the shader are illuminated based on the computer-generated lighting source, thereby illuminating one or more real-world objects in the environment with direct lighting from the computer-generated lighting source and reflected and refracted lighting from surfaces in the environment.

BACKGROUND Field

Aspects of the present disclosure relate to augmented reality displays,and more specifically to rendering lighting effects on real-worldobjects in an augmented reality display.

Description of the Related Art

Head-mounted augmented reality devices (also referred to as head-mounteddisplays, or “HMDs”) generally are display devices that are worn on auser's head and display, to the wearer of the device, an image of thereal-world environment in which the HMD is being used with additionalinformation superimposed on the image of the real-world environment.Some of these devices may be opaque to the outside world. Using a cameracoupled with the HMD, these devices generally capture images of thereal-world environment in which the HMD is being used and combine thesecaptured images with augmented reality content rendered in a layer witha transparent background (e.g., a layer with a transparent alphachannel). Other devices, known as see-through HMDs, may havesemi-transparent lenses and a projector configured to display augmentedreality content on the semi-transparent lenses, which may allow a userto view the real-world environment in which the HMD is being used withadditional content superimposed over the real-world environment.

The real-world environment in which an HMD is being used generallyincludes a variety of objects made from different materials. Some ofthese materials may have low reflectivity (e.g., having matte finishesor other non-reflective surfaces) and thus may reflect light in adiffused manner or not reflect light. Other surfaces in the real-worldenvironment, such as polished metal surfaces, glass, mirrors, and thelike may have highly reflective surfaces that directly reflect lightshining on the surface. An amount of light reflected by any object may,thus, differ based on how reflective each surface in the real-worldenvironment is, based on whether the object is within a line of sight ofa light source, the intensity of the light source, and the like.

In augmented reality systems, artificial, or computer-generated, lightsources may be digital objects overlaid on the environment in which theHMD is being used. For see-through HMDs, overlaying a lighting sourceover the real-world environment may result in a digital object beingdisplayed in the user's view of the real-world environment. However,rendering a lighting source and overlaying the lighting source in theuser's view of the real-world environment may not modify the user's viewof the real-world environment beyond displaying the lighting source inthe user's view of the real-world environment.

SUMMARY

One embodiment described herein is a method for rendering lightingeffects in an augmented reality display. The method generally includesoverlaying a shader on the augmented reality display. The shadergenerally corresponds to a three-dimensional geometry of an environmentin which the augmented reality display is operating, and the shadergenerally comprises a plurality of vertices forming a plurality ofpolygons. A computer-generated lighting source is introduced into theaugmented reality display. One or more polygons of the shader areilluminated based on the computer-generated lighting source, therebyilluminating one or more real-world objects in the environment withdirect lighting from the computer-generated lighting source andreflected and refracted lighting from surfaces in the environment.

Another embodiment described herein is a system for rendering lightingeffects in an augmented reality display. The system generally includes aprocessor and memory. The memory stores instructions that, when executedby the processor, performs an operation that generally includesoverlaying a shader on the augmented reality display. The shadergenerally corresponds to a three-dimensional geometry of an environmentin which the augmented reality display is operating, and the shadergenerally comprises a plurality of vertices forming a plurality ofpolygons. A computer-generated lighting source is introduced into theaugmented reality display. One or more polygons of the shader areilluminated based on the computer-generated lighting source, therebyilluminating one or more real-world objects in the environment withdirect lighting from the computer-generated lighting source andreflected and refracted lighting from surfaces in the environment.

Still another embodiment described herein is a head-mounted display forrendering and displaying an augmented reality experience. Thehead-mounted display generally includes a non-blocking display screenand a system for rendering content on the non-blocking display.Generally, the environment in which the head-mounted display isoperating is visible through the non-blocking display screen. The systemgenerally overlays a shader on the non-blocking display screen. Theshader generally corresponds to a three-dimensional geometry of anenvironment in which the augmented reality display is operating, and theshader generally comprises a plurality of vertices forming a pluralityof polygons. A computer-generated lighting source is introduced into theaugmented reality display. One or more polygons of the shader areilluminated based on the computer-generated lighting source, therebyilluminating one or more real-world objects in the environment withdirect lighting from the computer-generated lighting source andreflected and refracted lighting from surfaces in the environment.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited aspects are attained andcan be understood in detail, a more particular description ofembodiments described herein, briefly summarized above, may be had byreference to the appended drawings.

It is to be noted, however, that the appended drawings illustratetypical embodiments and are therefore not to be considered limiting;other equally effective embodiments are contemplated.

FIG. 1A is a block diagram of a head-mounted augmented reality displayin which a shader is used to display lighting effects on objects in thereal-world environment visible in an augmented reality display,according to one embodiment described herein.

FIG. 1B is a block diagram of an augmented reality display in which ashader generated by a remote system is used to display lighting effectson objects in the real-world environment, according to one embodimentdescribed herein.

FIG. 2 illustrates example operations for rendering lighting effects onreal-world objects in an augmented reality display using a shader and acomputer-generated light source, according to one embodiment describedherein.

FIG. 3 illustrates example operations for generating a shader used inrendering lighting effects on real-world objects in an augmented realitydisplay, according to one embodiment described herein.

FIG. 4 illustrates an example environment in which an augmented realitydisplay is operating and an example rendering of lighting effects on theenvironment, according to one embodiment described herein.

FIG. 5 illustrates an example system for rendering lighting effects onobjects in a real-world environment visible in an augmented realitydisplay, according to one embodiment described herein.

DETAILED DESCRIPTION

Embodiments of the present disclosure describe augmented reality systemsthat render, from computer-generated light sources displayed in anaugmented reality display, lighting effects on real-world objects in theenvironment in which the augmented reality display is being used. In oneembodiment, the augmented reality system includes a shader generatedfrom a three-dimensional geometry of the environment in which theaugmented reality display is being used. The shader generally includes aplurality of vertices connected to form a plurality of polygons. In someembodiments, the shader may have a flat texture, and the vertices may beunilluminated, such that in the absence of computer-generated lightsources, the shader is transparent to the user of the augmented realitydisplay. When computer-generated light sources are introduced into theaugmented reality display, vertices of the shader may be illuminated torender lighting effects on one or more polygons in the shadercorresponding to locations of different real-world objects.

By rendering lighting effects in a shader overlaid on a user's view ofthe real-world environment in which the user is wearing an augmentedreality display, embodiments described herein may generate scenes in anaugmented reality display that realistically model light interaction(e.g., reflection, refraction, absorption, diffusion, etc.) ofcomputer-generated light sources on real-world objects. In someembodiments, the shader may be used to make additional modifications tothe user's view of the real-world environment. For example, shaders canbe customized to compensate for a user's visual deficiencies so that theentirety of a scene is viewed through a color compensating filter thatallows the user to see the actual colors, or an approximation of theactual colors, of the real-world environment. Computer-generated lightsources may further be layered on top of a customized shader so that thelighting effects on real-world objects are rendered in a color-correctedmanner.

FIG. 1A illustrates a block diagram of an example head-mounted augmentedreality display 100A in which shaders are generated and used to renderartificial lighting effects on real-world objects in the augmentedreality display, according to one embodiment described herein.Generally, head-mounted augmented reality display 100A may be asee-through display through which a user of the augmented realitydisplay 100A can directly view the real-world environment in which theuser is using the head-mounted augmented reality display 100A (asopposed to a digital image of the real-world environment). Thehead-mounted augmented reality display 100A may overlay the user'sperspective of the real-world environment with a computer-generateddisplay of one or more digital objects. As illustrated, augmentedreality display 100A includes a processing system 110, a camera 120, adisplay 130, and a lens 140.

Processing system 110 is generally configured to generate a shader fromone or more images of the real-world environment in which augmentedreality display 100A is operating, render the generated shader ondisplay 130, and render digital objects and lighting effects created bythe rendered digital objects using the generated shader. As discussedabove, by rendering lighting effects using the generated shader,computer-generated light sources that exist only in the augmentedreality space can display lighting effects over real-world objects inthe environment. By displaying lighting effects over real-world objects,processing system 110 can render a more realistic augmented realityexperience than an augmented reality experience in which digital objectsare overlaid on a real-world environment without rendering lightingeffects on real-world objects.

As illustrated, processing system 110 includes an environment propertyidentifier 112, shader generator 114, light source generator 116, andscene renderer 118. Environment property identifier 112 uses one or moreimages of the real-world environment to determine the geometry of thereal-world environment and other properties that may be used inrendering lighting effects various objects in the real-worldenvironment. The one or more images of the real-world environment may bereceived as still images, sequences of still images, or video feeds.Generally, environment property identifier 112 receives a plurality ofimages from camera 120, which may be connected to processing system 110via a cable or wirelessly, and can use information such as focusdistance, depth detection, and other distance-related attributes todetermine the geometry of the real-world environment. For example,augmented reality distance determination techniques can be used todetermine the distance between different objects in the real-worldenvironment, which may affect an amount of light reflecting on anobject, as light intensity may decrease as distance between objectsincreases. In some embodiments, system 110 can identify and/or confirmthe presence of various objects within the real-world environment usingvarious tagging techniques, such as based on information broadcastwirelessly from various objects (e.g., through NFC, Bluetooth, or otherwireless communications protocols), scanning of barcodes (e.g., QRcodes), and the like.

In some embodiments, room property identifier 112 can control camera 120and lighting components associated with camera 120 to obtain images ofthe real-world environment with additional illumination directed toreal-world objects in the environment and without additionalillumination directed to the real-world objects. Based on differences inpixel brightness between images of a real-world object withoutadditional illumination and with additional illumination, room propertyidentifier 112 can determine the reflectivity of a surface. For example,highly reflective surfaces, such as mirrors, polished metal, and thelike may have a large difference in pixel brightness between images withand without additional illumination, while less reflective surfaces,such as surfaces with matte finishes, cloth surfaces (e.g., drapery,cloth upholstered furniture, etc.), walls, and the like may have smallerdifferences in pixel brightness between images with and withoutadditional illumination. In some embodiments, reflectivity of a surfacemay be determined independently of color information. To do so, imagesmay be converted from a color image to a grayscale image (e.g., from animage with 8-bit red, green, and blue channels to an image with a single8-bit greyscale channel), and pixel locations corresponding to anilluminated and unilluminated object may be identified. The differencebetween the brightness of these pixel locations may be mapped to anamount of reflectivity to be implemented by the shader, as discussed infurther detail below. Generally, large differences in object brightnessbetween images with and without additional illumination may indicatethat an object is highly reflective, and thus that a shader should beconfigured to reflect a significant amount of light off of that object.Meanwhile, smaller differences in object brightness between images withand without additional illumination may indicate that an object has lowreflectance or is not reflective, and thus that a shader should beconfigured to reflect a small amount of light off of that object.Further, information about the total area illuminated by a light sourcemay be gathered from images of an object with and without additionalillumination. For example, given a light source with a defined size, thesize of the illuminated area of the object may be substantially similarto the size of the light source for highly reflective objects.Meanwhile, the size of the illuminated area of the object may be larger,but less intense, for objects that are not highly reflective (e.g.,objects with a matte or antiglare finish). The size of the illuminatedarea of the object may thus be used alone or in conjunction withbrightness information to determine the reflectivity of a surface andcorrespondingly how a shader is configured to reflect light from acomputer-generated light source.

In some embodiments, room property identifier 112 can additionally oralternatively use object identification techniques to identify an objectand search for information about that object (or the surface materialsof that object) from a remote data repository (not illustrated). Roomproperty identifier 112 can identify an object as a general class ofobject (e.g., a mirror, a desk, a chair, etc.) and can use portions ofan image to identify a type of material the object is made from. Forexample, suppose that room property identifier 112 identifies an objectas an upholstered chair. Room property identifier can then select aportion of an image including a portion of the upholstered chair andsubmit the portion of the image to a remote service to determine thetype of material the chair is made from. Using the information about thetype of object and the type of material, room property identifier 112can search a lookup table of predefined reflectivity information todetermine the reflectivity of the object for use in generating a shaderthat renders lighting effects on real-world objects, as discussed infurther detail below.

Shader generator 114 uses the room geometry and reflectivity informationto generate a shader used by head-mounted augmented reality display 100Ato render lighting effects on real-world objects. Generally, shadergenerator 114 can use the determined room geometry to generate shadervertices and polygons that are used to overlay illumination effects onreal-world objects in the real-world environment based on theintroduction of a computer-generated light source in the head-mountedaugmented reality display 100A. Generally, vertices may be positionedsuch that any number of vertices can be connected by edges, and multipleedges may form polygons in the shader. Generally, the shader verticesand polygons may be generated by shader generator 114 to account for theshape, size, and contours of real-world objects in the real-worldenvironment that the augmented reality display 100A is operating.

In some embodiments, the shader generated by shader generator 114 may beconfigured with a plain white texture and vertices colored substantiallysimilar to the tint of lens 140 of augmented reality display 100A.Generally, by generating the shader with a plain white texture and lenstint-colored vertices, the shader may be configured to be transparent orsubstantially transparent to a user of augmented reality display 100A.To compensate for differences in the reflectance of various surfaces inthe real-world environment, polygons corresponding to different surfacesmay be associated with different reflectivity properties so that when anartificial light source is introduced into a view of the real-worldenvironment, the light reflected from each surface corresponds to thereflectivity of a surface (e.g., so that light is reflected at a higherintensity from more reflective surfaces and at a lower intensity fromless reflective surfaces). In some embodiments, the vertices may becolored to effectively apply a color compensating filter to theaugmented reality display 100A. For example, to compensate for a userdeficiency in perceiving a particular color, the vertices may be coloredaccording to a color that filters out colors that a user is unable tosee. In still further embodiments, the shader generated by shadergenerator 114 may be configured with various colors of vertices so as toapply scenario-specific color tints to the real-world environment inwhich augmented reality display 100A is being used.

In some embodiments, room property identifier 112 and shader generator114 may periodically scan the real-world environment in which theaugmented reality display 100A is operating to identify changes to thereal-world environment that are not reflected in the configuration ofthe generated shader. For example, these changes may include theintroduction of new objects into the real-world environment, opening andclosing of doors, windows, and other light entry apertures in thereal-world environment, changes in the reflectivity of surfaces in thereal-world environment (e.g., previously clear windows or mirrorsbecoming frosted over), and the like. In some embodiments, these changesmay further include the introduction of fog, mist, or other weathereffects that may change how light interacts with objects in thereal-world environment. By periodically scanning the real-worldenvironment and updating the shader based on each new scan of thereal-world environment, room property identifier 112 and shadergenerator 114 can apply lighting effects to a shader corresponding tothe current state of the real-world environment rather than an outdatedstate of the real-world environment. To use one example, suppose a solidwooden door with a minimally reflective surface is opened, occludingpart of a chromed surface in the real-life environment. A previousshader configured for the previous state of the real-world environment(i.e., a state in which the solid wooden door was closed) may result ina lighting effect that applies the reflectivity of the chromed surfaceto the wooden door. By updating the shader to account for the openeddoor (and therefore the portion of the chromed surface obscured by theopened door), reflections may not be generated on surfaces that are nolonger visible in the real-world environment.

After shader generator 114 generates the shader, shader generator 114deploys the shader to scene renderer 118 for rendering and display to auser of the augmented reality display 100A. As discussed, in the absenceof artificial light sources included in a view of the real-worldenvironment, the shader rendered over the user's view of the real-worldenvironment generally remains transparent to the user. Thus, in theabsence of artificial light sources, a user should not be aware thatgraphical elements are being rendered on display 130 and overlaid on theuser's view of the real-world environment.

Light source generator 116 may be a component of a game or otherapplication that introduces lighting effects into a user's view of thereal-world environment in which augmented reality display 100A is beingused. Generally, light source generator 116 may define a position withindisplay 130 at which a light source is to be introduced and variousproperties of the light source. For example, these properties mayinclude a shape and size of the light source, a color of light, anintensity of the light, and so on. In some embodiments, light sourcegenerator 116 may generate lighting effects for a moving virtual lightsource in the augmented reality display. These moving light sources maybe user-controlled (e.g., a laser sword controlled by a user in a game)or may be computer-controlled.

Light source generator 116 generally provides information about thegenerated light source to scene renderer 118 for use in adding lightingeffects to the user's view of the real-world environment based on thegenerated shader. In some embodiments, for a light source overlaid on agiven position in the real-world environment, scene renderer 118 canidentify the vertices to illuminate to add lighting effects to thereal-world environment. The vertices may be illuminated, for example,based on a color and intensity of the generated light source such thatlighting effects are rendered with an intensity commensurate with thereflectivity of each service identified in the real-world environmentand the distance between the user and each object in the real-worldenvironment. Generally, for a given surface, reflectivity and otherlight effects may be modeled with less intensity (e.g., less brightness)as the distance between the generated light source and an object in thereal-world increases. Further, scene renderer 118 may be configured todiffuse (e.g., spread) the lighting effects from a generated lightsource over larger areas based on the reflectivity properties of a givensurface of an object in the real-world environment. For example, objectsthat have low reflectance surfaces may result in scene renderer 118overlaying a larger, but less intense, illumination over the object tosimulate the diffusion of reflected light that would occur on such asurface in real life. Meanwhile, objects with high reflectance surfacesmay result in scene renderer 118 overlaying a smaller, but more intense,illumination over the object to simulate the reflection (e.g., as apoint source) of light that would occur on a high reflectance surface inreal life. In some embodiments, scene renderer 118 may additionally usereflectivity information to determine and render reflected and refractedlight effects from one object to another object. For example, objectsdetermined to be highly reflective may bounce light onto another object,and the lighting effects rendered on the other object may be acombination of illumination from the generated light source andreflected light from reflective objects in the real-world environment.

In some embodiments, illumination effects may be added to the real-worldenvironment on a per-shader-pixel-fragment basis in lieu of or inconjunction with the vertex lighting techniques discussed above. Addingillumination effects for a shader pixel fragment may providefiner-grained control over the addition of illumination effects toreal-world objects in the real-world environment. For example, theaddition of illumination effects on a per-shader-pixel-fragment basismay allow for localized generation of lighting effects that may not beconstrained by the size and shape of a polygon defined by connectionsbetween vertices in the shader. The addition of these illuminationeffects using shader pixel fragments may be used, for example, to addillumination effects to different edges of real-world objects (e.g., toadd blooming or glare effects to edges of a real-world object based onthe reflectivity of the real-world object), to render point-sourcereflections to a real-world object where the reflection is smaller thana polygon in the shader, and the like.

In some embodiments, illumination effects rendered using the shader maymake physically bright surfaces optically darker and physically darkersurfaces optically lighter. By rendering lighting effects as opticallylighter for physically darker surfaces, an intensity of the lightingeffects rendered on display 130 may blend with a color of lens 140 suchthat lighting effects over unilluminated or minimally illuminatedreal-world objects appear transparent to a wearer of augmented realitydisplay 100A. Meanwhile, more intense lighting effects can be renderedas darker areas on display 130 (e.g., rendered as areas with moreintense coloration and/or alpha channels that are opaque or near-opaqueso that lighting effects are visible over the corresponding real-worldobject through lens 140.

Generally, the addition of illumination effects to real-world objects inthe environment in which the head mounted display 100A is operating maybe rendered such that adjustments to lighting effects are made withrespect to a global perceptual anchor object. The global perceptualanchor may be monitored periodically (e.g., based on the objects visibleto the user through lens 140. In some embodiments, the global perceptualanchor may be an arbitrarily selected object that is not expected tohave significant brightness changes. By rendering lighting effects withrespect to the global perceptual anchor object, scene renderer 118 neednot attempt to render lights and colors that are out of range, asilluminations may be rendered so that the reflections are perceived ashaving sufficient brightness using small changes to the color andintensity of the rendered lighting effects.

Camera 120 is illustrated to be representative of various still or videoimage capture devices that may be used to capture images of thereal-world environment in which an augmented reality device 100Aoperates. As discussed above, information about the real-worldenvironment captured by camera 120 may be fed into environment propertyidentifier 112 of processing system 110, where images are processed toidentify the size and light reflectivity characteristics of objects inthe real-world environment. In some embodiments, camera 120 may bemounted on a moving platform that moves to reflect the eye position ofthe user of augmented reality device 100A. Captured images of thereal-world environment (or portion of the real-world environment) inwhich the user is using the augmented reality device 100A may be used toadjust the portion of a shader overlaid on the user's view of thereal-world environment so that lighting effects are rendered on theappropriate objects and within the boundaries of each object asperceived by the user of the augmented reality device 100A.

Display 130 generally receives a rendered scene from scene renderer 118for display to a user of the augmented reality device 100A. To allowaugmented reality device 100A to function as a see-through device,display 130 may be a projector configured to project an image on lens140. Generally, by projecting an image generated from lighting effectsapplied to a shader with transparent vertices, lighting effects may beoverlaid on the user's view of the real-world environment so thatreflections in the real-world environment are illuminated by the shaderand other areas remain unilluminated so that the real-world objectsremain visible to the user. When no artificial lighting sources arerendered in the scene, the shader may appear transparent to the user ofthe augmented reality device 100A so that the user can view thereal-world environment without modification. Lens 140 may be an opticalcombiner configured to combine a projected image from display 130 withthe real-world environment so that the combination of the real-worldenvironment and the rendered artificial lighting sources are visible tothe user of the augmented reality device 100A.

FIG. 1B illustrates an augmented reality device 100B in which a shaderis generated on a remote processing system an deployed to a processor ona head mounted display unit to render lighting effects on real-lifeobjects. As illustrated, augmented reality device 100B includes a camera120, display 130, lens 140, remote processing system 150, and headmounted display (HMD) processor 160. A head mounted display unit worn bya user of augmented reality device 100B may include camera 120, display130, lens 140, and HMD processor 160 and may be connected via a networkconnection to remote processing system 150.

Remote processing system 150 is illustrative of various computingdevices that can receive data from a client device and process thereceived data to generate a shader for use in augmenting a real-liveenvironment with lighting effects from a computer-generated lightsource. As illustrated, remote processing system 150 includesenvironment property identifier 112 and shader generator 114. Remoteprocessing system 150 generally receives images captured of theenvironment in which the augmented reality device 100B is operating fromcamera 120 for processing. Remote processing system 150 may receivethese images wirelessly over a personal area network, a wireless localarea network (e.g., an 802.11 WiFi network connection), or a wide areanetwork (e.g., a cellular network) from camera 120. A wirelessconnection between remote processing system 150 and HMD processor 160may generally be a low-latency, high bandwidth wireless connection thatallows large amounts of data to be transmitted in short amounts of time.As remote processing system 150 receives images of the real-worldenvironment in which augmented reality device 100B is operating,environment property identifier 112 can identify the shape and size ofvarious objects in the real-world environment, as well as reflectivityproperties of different surfaces in the real-world environment, asdiscussed above. Based on the identified shape and size of variousobjects in the real-world environment, shader generator 114 generates ashader with vertices having a color substantially similar to that of thelens 140 through which a user of the augmented reality device 100B viewsthe real-world environment and a flat, white texture that isunilluminated in the absence of artificial light sources overlaid in thereal-world environment. Once generated, shader generator 114 cantransmit the generated shader definition to HMD processor 160 for use inrendering artificial light sources and reflections and overlaying theserendered light sources and reflections over real-world objects inaugmented reality device 1006.

In some embodiments, room geometry identifier 112 need not identifyproperties of a room that may be used in rendering artificial lightsources and reflections from these artificial light sources on objectsin the real world. Room geometry identifier 112 can capture one or moreimages of the environment in which the display is being used and mayquery a data repository for information about the real-worldenvironment. If a shader has previously been generated for a givenenvironment, room geometry identifier 112 can obtain the matching shaderfrom a remote destination and use the retrieved shader to renderlighting effects in the augmented reality display 1006. As discussed,this information may include distance and reflection/refractioninformation so that the correct objects are illuminated when anartificial lighting source is introduced on screen 130.

HMD processor 160 is generally illustrative of various computing devicescoupled with a head-mounted display that can generate artificiallighting sources on display 140 for display to a user, according to oneembodiment. HMD processor, as illustrated, generally includes a lightsource generator 118 and a scene renderer 118. As discussed above, lightsource generator 118 is generally configured to render one or moreartificial light sources for display over the real-world environment inwhich augmented reality device 1006 is operating, and scene renderer 118is configured to use information about the rendered artificial lightsources, the shader generated from the determined geometry of thereal-world environment, and reflectivity information about each surfacein the real-world environment to render light reflections and otherlighting effects over various objects in the real-world environment. Therendered scene generated by scene renderer 118 may be transmitted todisplay 130, which projects the rendered scene onto lens 140 to combinethe generated lighting effects with the real-world environment.

FIG. 2 illustrates example operations that may be performed by anaugmented reality device to render lighting effects on real-worldobjects in the environment in which the augmented reality deviceoperates. As illustrated, operations 200 begin at block 210, where theaugmented reality device generates a shader corresponding to athree-dimensional geometry of an environment in which the augmentedreality display is operating. As discussed, in some embodiments, theaugmented reality device can generate the shader by capturing one ormore images of the real-world environment and using distance informationderived from the captured images to identify surfaces in the real-worldenvironment. Using the captured images, the augmented reality device canidentify boundaries between different shapes in the real-worldenvironment (corresponding to different surfaces) and can generate andconnect vertices to form a vector map of a plurality of polygons. Theshader may have a flat, uncolored texture, and the vertices may becolored substantially similarly to the color of the lenses of theaugmented reality device, so that the shader is transparent to the userwhen computer-generated artificial light sources have not been overlaidon the real-world environment.

In some embodiments, the augmented reality device can generate a shaderbased on an a priori known arrangement of objects in the real-worldenvironment. For example, if an augmented reality device is used in oneof a set of known environments (e.g., in an entertainment setting wherethe architecture of the rooms in which the augmented reality device isto be used are known), the augmented reality system can use imagescaptured by a camera of the augmented reality device to identify theroom and retrieve a shader model corresponding to the room from a datarepository.

At block 220, the augmented reality system overlays the shader on theaugmented reality device. To overlay the shader on the augmented realitydisplay, the augmented reality device can render a scene using theshader and use the display to project the rendered scene on top of thelenses of the augmented reality system. By projecting the rendered sceneon top of the lenses, the system can add rendered content to the user'sview of the real-world environment. Generally, when the augmentedreality device has not rendered any artificial light sources forinclusion in the user's view of the real-world environment, the shadermay be transparent to the user and thus may not modify the user's viewof the real-world environment. In some embodiments, however, asdiscussed above, the shader may be customized to alter the user's viewof the real world. For example, the shader may be customized tocompensate for color vision deficiencies (e.g., red-greencolorblindness, blue-yellow colorblindness, etc.) or to introduce a tintto the user's view of the real world environment in which the augmentedreality device is being used.

At block 230, the augmented reality device introduces acomputer-generated lighting source into the environment viewed throughthe augmented reality display. The computer-generated lighting source isgenerally a lighting source that is not present in the real-worldenvironment and is rendered as an object overlaid on the real-worldenvironment. Computer-generated lighting sources may be generated asstatic or moving objects and with static or dynamic lighting properties(e.g., a lighting source with a consistent color and/or brightness or alighting source with changing color and/or brightness).

At block 240, the augmented reality device illuminates one or morepolygons of the shader to illuminate one or more real-world objects inthe real-world environment based on the computer-generated lightingsource introduced into the environment at block 230. Generally, byilluminating one or more polygons of the shader and displaying theilluminated polygons over real-world objects in the user's view of thereal-world environment, a user may perceive that real-world objects areilluminated with direct and indirect lighting. Generally, inilluminating polygons of the shader, the augmented reality device canuse distance information between the augmented reality device andobjects in the real-world environment, reflectivity information abouteach surface in the real-world environment, and the brightness of thecomputer-generated lighting source to determine the intensity and sizeof the lighting effects rendered over each real-world object in thereal-world environment.

To render a lighting effect, a vector representing the characteristicsof the computer-generated lighting source can be introduced into theshader. For each polygon in the shader, the augmented reality device cancalculate the appearance of the polygon to overlay a lighting effect ona real-world object in the real-world environment. Generally, the areailluminated by the object may be determined based on the recordedreflectivity of each surface, as previously discussed, so thatreflective surfaces have a brighter appearance than less reflectivesurfaces. Distance calculations between the computer-generated lightsource (which may be assumed to be at the same position as the user ofthe augmented reality device or at a different position) may be used todetermine the size and intensity of an illumination effect, asreflections of objects may decrease in size and intensity as thedistance between a light source and an object increases.

In some embodiments, the computer-generated lighting source can berendered outside of the user's view of the real-world environment. Torender a lighting source outside of the user's view of the real-worldenvironment, a lighting source may be introduced at a pixel locationoutside of the resolution of the display components of the augmentedreality device. For example, a lighting source may be introduced at anegative pixel location to generate a lighting effect from the left sideof the display (horizonally) or the bottom of the display (vertically)or at a pixel location in excess of the horizontal resolution (i.e., ahorizontal pixel location greater than or equal to 1920, on a 1080pdisplay) or vertical resolution (i.e., a vertical pixel location greaterthan or equal to 1080, on a 1080p display) in order for the lightingsource to not be visible to the user of the augmented reality device.Lighting effects may be rendered based on an assumed location anddirectionality of the lighting source so that real-world objects can beilluminated from computer-generated light sources that are not visibleto the user.

FIG. 3 illustrates example operations that may be performed by anaugmented reality device to generate a shader used to render lightingeffects on real-world objects in which the augmented reality deviceoperates. As illustrated, operations 300 begin at block 310, where theaugmented reality device captures one or more images of an environmentin which an augmented reality display is operating. The augmentedreality device can capture these images using a camera integral to orotherwise coupled with the augmented reality device and can includedistance information for each object in the real-world environment. Insome embodiments, multiple images may be captured of a particularposition in the real-world environment that the augmented reality deviceis being used in. Some of these images may be captured using available(ambient) light, and some of these images may be captured usingadditional illumination (e.g., a flash or continuous light source). Asdiscussed, capturing images with and without additional illumination mayallow the augmented reality device to capture additional informationthat may be used in rendering illuminations over real-world objects sothat objects with more reflective surfaces have lighting effectsrendered with more intensity than objects with less reflective surfaces.

At block 320, the augmented reality device determines athree-dimensional geometry of the real-world environment in which theaugmented reality display is operating based on the captured one or moreimages. The three-dimensional geometry may be determined, for example,based on contours of objects detected in the captured images, distanceinformation encoded in the image or otherwise captured by the camera,edges identified in the captured images, and the like. In someembodiments, some information about the three-dimensional geometry ofthe real-world environment may be obtained from a remote source. Thisinformation may include, for example, information about known roomdimensions and other architectural features that may influence thethree-dimensional geometry of the real-world environment. In someembodiments, where the augmented reality device is used in an openenvironment (e.g., outdoors), the three-dimensional geometry may bedetermined for objects within a threshold distance from the augmentedreality display. Beyond this threshold distance, which may be aninfinity focal distance of a camera coupled with the augmented realitydevice, objects may be assumed to have a two-dimensional appearance ornot be relevant to the rendering of artificial lighting effects overreal-world objects.

At block 330, the augmented reality device generates polygons of ashader based on the determined three-dimensional geometry of theenvironment. Polygons of the shader may be defined as a set of connectedvertices that forms a shape in space. These polygons may be defined, forexample, as shapes that account for the contours of each real-worldobject in the real-world environment. The polygons of the shader may bedefined with a flat white texture so that when a polygon is illuminated,the polygon takes on the color of the light reflected on a surface. Thatis, if a light source is generated with a particular color andintroduced into the user's view of the real-world environment,reflections of the light source from real-world objects are also someshade of that particular color (dependent on the reflectivity and lightdiffusion properties of an object, where more reflective objects havelighting effects that are brighter and closer to the particular color ofthe light source and less reflective objects have lighting effects thatare a dimmer or lighter hue of the particular color of the lightsource).

At block 340, the system colors vertices in the shader such that theshader appears transparent in the augmented reality display in theabsence of computer-generated lighting effects. Generally, the verticesin the shader may be colored with a color substantially similar to thatof the lenses of the augmented reality display so that the shader isvisually transparent to the user. As discussed, when artificial lightsources are generated and overlaid on the user's view of the real-worldenvironment, the shader may be illuminated based on the location of theartificial light source, the distance between the artificial lightsource and objects in the real-world environment, and the lightingproperties of the artificial light source so that illuminated polygonsin the shader are visually combined with the real-world objects tosimulate the appearance of reflections or other lighting effects in theaugmented reality view of the real-world environment.

FIG. 4 illustrates images of a room with and without the application oflighting effects from a shader, according to embodiments describedherein. Image 402 depicts a real-world environment in which an augmentedreality device may be used to view the real-world environment andcomputer-generated content overlaid on the real-world environment.Generally, in the absence of additional computer-generated lightsources, a user may see an image substantially similar to image 402through the augmented reality device. As discussed, the shader generatedfrom the three-dimensional geometry of the room may have a flat whitecolor and vertices with the same or substantially similar color as thelenses of the augmented reality device so that the shader appearstransparent to the user when rendered and overlaid on the user's view ofthe real-world environment in the absence of computer-generated lightsources.

Image 404 depicts an augmented reality view of the environmentillustrated in image 402 in which a light source is added to thereal-world environment to introduce illumination effects to thereal-world environment. In this example, the light source is positionedoutside of the user's view of the real-world environment illustrated inimage 402 and is directed towards the map on the wall on the left sideof the image. Because lighting effects are additive in an opticalsee-through augmented reality device, the polygons of the shadercorresponding to the location of the map in the user's view of thereal-world environment may be illuminated based on the color andintensity of the computer-generated light source. The addition of acomputer-generated light source and the illumination effects of thecomputer-generated light source on real-world objects may be used tosimulate reflections on different surfaces caused by these lightsources. In some embodiments, as illustrated in image 404, the additionof illumination effects from a computer-generated light source may beused to change the appearance of a real-world object displayed to a userof the augmented reality display (e.g., to make hidden content visible,such as a hidden treasure map in a standard map).

As illustrated, image 404 may be darker when viewed through lenses of ahead-mounted augmented reality device than image 402. By darkening theuser's view of the real-world environment prior to adding illuminationeffects to the real-world environment, augmentation of the real-worldenvironment with illumination effects may allow for illumination effectsto be added that is consistent with the relative illumination of thesurfaces of real-world objects in the real-world environment. Darkeningimage 404 may thus allow for illumination effects to be overlaid overreal-world objects in the real-world environment without clipping orrendering the lighting effects with excessive intensity.

FIG. 5 illustrates an augmented reality system 500 on which shaders aregenerated from room geometry information and used to overlay lightingeffects on real-world objects. Augmented reality system 500 may berepresentative, for example, of augmented reality devices 100A or 100Billustrated in FIG. 1A or 1B.

As illustrated, augmented reality system 500 includes a centralprocessing unit (CPU) 502, one or more I/O device interfaces 504 thatmay allow for the connection of various I/O devices 515 (e.g.,keyboards, displays, mouse devices, pen input, etc.) to the server 500,network interface 506 through which server 500 is connected to network590 (which may be a local network, an intranet, the internet, or anyother group of computing devices communicatively connected to eachother), a memory 508, storage 510, and an interconnect 512.

CPU 502 may retrieve and execute programming instructions stored in thememory 508. Similarly, the CPU 502 may retrieve and store applicationdata residing in the memory 508. The interconnect 512 transmitsprogramming instructions and application data, among the CPU 502, I/Odevice interface 504, network interface 506, memory 508, and storage510.

CPU 502 is included to be representative of a single CPU, multiple CPUs,a single CPU having multiple processing cores, and the like.

Memory 508 is representative of a volatile memory, such as a randomaccess memory, or a nonvolatile memory, such as nonvolatile randomaccess memory, phase change random access memory, or the like. Asillustrated, memory 508 includes an environment property identifier 520,shader generator 530, light source generator 540, and a scene renderer550. Environment property identifier 520 generally receives one or moreimages of the real-world environment in which augmented reality system500 is being used to identify the three-dimensional geometry of theenvironment in which the augmented reality system 500 is being used,including information about different surfaces and different objects inthe real-world environment. In some embodiments, as discussed above, theimages may be captured with and without additional illumination, whichmay allow environment property identifier 520 to capture additionalinformation about surfaces in the real-world environment that can beused to render reflected lighting effects over real-world objects.

Shader generator 530 generally uses the room geometry information andother information about the real-world environment identified byenvironment property identifier 520 to generate a shader correspondingto the real-world environment. Generally, the shader includes aplurality of polygons formed from a plurality of connected vertices,with each polygon representing a portion of an object in the real-worldenvironment. The shader may be configured with a plain white texture andvertices colored substantially similar to a lens of augmented realitysystem 500 so that the shader appears transparent to the user.

Light source generator 540 generally generates artificial light sourcesand positions the generated light sources in the user's view of thereal-world environment to introduce virtual objects into the user's viewof the real-world environment. These artificial light sources may bestationary or may move throughout the user's view of the real-worldenvironment, and the lighting properties of these artificial lightsources may be arbitrarily defined so that a light can have a consistentor changing brightness, color, direction, and the like.

Scene renderer 550 uses information about the light sources generated bylight source generator 540, environment property information identifiedby environment property identifier 520, and the shader generated byshader generator 530 to render lighting effects on real-world objects inthe user's view of the real-world environment through augmented realitysystem 500. Generally, real-world objects may be illuminated byreflections determined based on the distance from a generated lightsource to an object, the reflectivity of the object, the brightness ofthe generated light source, and other relevant information. Polygonscorresponding to objects that are illuminated may become visible to theuser through a display connected to or integral with augmented realitysystem 500 so that the illuminated polygons and the object are combinedto generate the user's view of the object.

In the current disclosure, reference is made to various embodiments.However, it should be understood that the present disclosure is notlimited to specific described embodiments. Instead, any combination ofthe following features and elements, whether related to differentembodiments or not, is contemplated to implement and practice theteachings provided herein. Additionally, when elements of theembodiments are described in the form of “at least one of A and B,” itwill be understood that embodiments including element A exclusively,including element B exclusively, and including element A and B are eachcontemplated. Furthermore, although some embodiments may achieveadvantages over other possible solutions or over the prior art, whetheror not a particular advantage is achieved by a given embodiment is notlimiting of the present disclosure. Thus, the aspects, features,embodiments and advantages disclosed herein are merely illustrative andare not considered elements or limitations of the appended claims exceptwhere explicitly recited in a claim(s). Likewise, reference to “theinvention” shall not be construed as a generalization of any inventivesubject matter disclosed herein and shall not be considered to be anelement or limitation of the appended claims except where explicitlyrecited in a claim(s).

As will be appreciated by one skilled in the art, embodiments describedherein may be embodied as a system, method or computer program product.Accordingly, embodiments may take the form of an entirely hardwareembodiment, an entirely software embodiment (including firmware,resident software, micro-code, etc.) or an embodiment combining softwareand hardware aspects that may all generally be referred to herein as a“circuit,” “module” or “system.” Furthermore, embodiments describedherein may take the form of a computer program product embodied in oneor more computer readable medium(s) having computer readable programcode embodied thereon.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for embodiments of thepresent disclosure may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

Aspects of the present disclosure are described herein with reference toflowchart illustrations or block diagrams of methods, apparatuses(systems), and computer program products according to embodiments of thepresent disclosure. It will be understood that each block of theflowchart illustrations or block diagrams, and combinations of blocks inthe flowchart illustrations or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe block(s) of the flowchart illustrations or block diagrams.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other device to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the block(s) of the flowchartillustrations or block diagrams.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other device to cause aseries of operational steps to be performed on the computer, otherprogrammable apparatus or other device to produce a computer implementedprocess such that the instructions which execute on the computer, otherprogrammable data processing apparatus, or other device provideprocesses for implementing the functions/acts specified in the block(s)of the flowchart illustrations or block diagrams.

The flowchart illustrations and block diagrams in the Figures illustratethe architecture, functionality, and operation of possibleimplementations of systems, methods, and computer program productsaccording to various embodiments of the present disclosure. In thisregard, each block in the flowchart illustrations or block diagrams mayrepresent a module, segment, or portion of code, which comprises one ormore executable instructions for implementing the specified logicalfunction(s). It should also be noted that, in some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the Figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order or out of order, dependingupon the functionality involved. It will also be noted that each blockof the block diagrams or flowchart illustrations, and combinations ofblocks in the block diagrams or flowchart illustrations, can beimplemented by special purpose hardware-based systems that perform thespecified functions or acts, or combinations of special purpose hardwareand computer instructions.

While the foregoing is directed to embodiments of the presentdisclosure, other and further embodiments of the disclosure may bedevised without departing from the basic scope thereof, and the scopethereof is determined by the claims that follow.

What is claimed is:
 1. A method for rendering lighting effects in anaugmented reality display, comprising: overlaying, on an augmentedreality display, a shader corresponding to a three-dimensional geometryof an environment in which the augmented reality display is operating,the shader comprising a plurality of vertices forming a plurality ofpolygons and a flat texture on which lighting effects can be rendered;introducing a computer-generated lighting source into the augmentedreality display; and illuminating one or more polygons of the shader,thereby illuminating one or more real-world objects in the environmentwith direct lighting from the computer-generated lighting source andreflected and refracted lighting from surfaces in the environment. 2.The method of claim 1, wherein the vertices of the shader have a colorcorresponding to a color of a display screen of the augmented realitydisplay such that the shader, when unilluminated by thecomputer-generated lighting source, appears transparent.
 3. The methodof claim 1, further comprising: capturing the three-dimensional geometryof the environment in which the augmented reality display is operatingbased on a scan of the environment in which the augmented realitydisplay is operating; and generating the shader based on the capturedthree-dimensional geometry of the environment, wherein vertices in theshader correspond to edges of surfaces detected in the three-dimensionalgeometry of the environment.
 4. The method of claim 3, furthercomprising: measuring reflectivity of the surfaces in the environment inwhich the augmented reality display is operating, wherein the shadercomprises a plurality of vectors defining light reflectivity valuescorresponding to a respective surface in the environment based on themeasured reflectivity of the respective surface.
 5. The method of claim4, wherein illuminating one or more polygons of the shader comprisesilluminating one or more real-world objects in the environment in whichthe augmented reality display is operating such that illuminationsagainst objects with higher reflectivity surfaces appears brighter thanilluminations against objects with lower reflectivity surfaces, whereinilluminating the one or more real-world objects comprises decreasingvalues of one or more luminance channels for illuminated objects andincreasing values of the one or more luminance channels forunilluminated objects such that illuminations against objects areperceived consistently through a tinted see-through optical display. 6.The method of claim 3, wherein generating the shader based on thecaptured three-dimensional geometry of the environment comprisestransmitting one or more images of the environment to a remote systemvia a low latency, high bandwidth network connection.
 7. The method ofclaim 1, wherein the three-dimensional geometry of the environmentcomprises a predefined three-dimensional model of the environmentgenerated according to a known architectural layout of the environmentand reflectivity of surfaces in the environment.
 8. The method of claim1, wherein vertices of the shader are colored to compensate for colorvision deficiencies of a user.
 9. The method of claim 1, furthercomprising: updating the three-dimensional geometry of the environmentbased on a detected change in the environment.
 10. The method of claim9, wherein updating the three-dimensional geometry of the environmentcomprises: measuring reflectivity of one or more changed surfacesidentified in the detected change in the environment; and updating oneor more vectors in the shader based on the measured reflectivity of theone or more changed surfaces.
 11. A system, comprising: a processor; anda memory having instructions stored thereon which, when executed by theprocessor, performs an operation for rendering lighting effects in anaugmented reality display, the operation comprising: overlaying, on anaugmented reality display, a shader corresponding to a three-dimensionalgeometry of an environment in which the augmented reality display isoperating, the shader comprising a plurality of vertices forming aplurality of polygons and a flat texture on which lighting effects canbe rendered; introducing a computer-generated lighting source into theaugmented reality display; and illuminating one or more polygons of theshader, thereby illuminating one or more real-world objects in theenvironment with direct lighting from the computer-generated lightingsource and reflected and refracted lighting from surfaces in theenvironment.
 12. The system of claim 11, wherein the vertices of theshader have a color corresponding to a color of a display screen of theaugmented reality display such that the shader, when unilluminated bythe computer-generated lighting source, appears transparent.
 13. Thesystem of claim 11, wherein the operation further comprises: capturingthe three-dimensional geometry of the environment in which the augmentedreality display is operating based on a scan of the environment in whichthe augmented reality display is operating; measuring reflectivity ofthe surfaces in the environment in which the augmented reality displayis operating; and generating the shader based on the capturedthree-dimensional geometry of the environment and the measuredreflectivity of the surfaces in the environment, wherein vertices in theshader correspond to edges of surfaces detected in the three-dimensionalgeometry of the environment and are based on the measured reflectivityof the surfaces in the environment.
 14. The system of claim 13, whereinilluminating one or more polygons of the shader comprises illuminatingone or more real-world objects in the environment in which the augmentedreality display is operating such that illuminations against objectswith higher reflectivity surfaces appears brighter than illuminationsagainst objects with lower reflectivity surfaces, wherein illuminatingthe one or more real-world objects comprises decreasing values of one ormore luminance channels for illuminated objects and increasing values ofthe one or more luminance channels for unilluminated objects such thatilluminations against objects are perceived consistently through atinted see-through optical display.
 15. The system of claim 13, whereingenerating the shader based on the captured three-dimensional geometryof the environment comprises transmitting one or more images of theenvironment to a remote system via a low latency, high bandwidth networkconnection.
 16. The system of claim 11, wherein the three-dimensionalgeometry of the environment comprises a predefined three-dimensionalmodel of the environment generated according to a known architecturallayout of the environment and reflectivity of surfaces in theenvironment.
 17. The system of claim 11, wherein vertices of the shaderare colored to compensate for color vision deficiencies of a user. 18.The system of claim 11, wherein the operation further comprises:updating the three-dimensional geometry of the environment based on adetected change in the environment.
 19. The system of claim 18, whereinupdating the three-dimensional geometry of the environment comprises:measuring reflectivity of one or more changed surfaces identified in thedetected change in the environment; and updating one or more vectors inthe shader based on the measured reflectivity of the one or more changedsurfaces.
 20. A head-mounted display for rendering augmented realityexperiences, comprising: a non-blocking display screen in which anenvironment in which the head-mounted display is visible through thenon-blocking display screen; and a system for rendering content on thenon-blocking display, the system being configured to: overlay, on thenon-blocking display screen, a shader corresponding to athree-dimensional geometry of an environment in which the augmentedreality display is operating, the shader comprising a plurality ofvertices forming a plurality of polygons and a flat texture on whichlighting effects can be rendered; rendering a computer-generatedlighting source on the non-blocking display; and illuminating one ormore polygons of the shader, thereby illuminating one or more real-worldobjects in the environment with direct lighting from thecomputer-generated lighting source and reflected and refracted lightingfrom surfaces in the environment rendered on the non-blocking display.